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I . INTRODUCTION 

PURPOSE 

The purpose of this research project was to identify and 
assess the quality assurance methods used by providers of financial 
management education and training courses and programs offered 
throughout the DoD. On July 3, 1991, the DoD Comptroller issued a 
memorandum entitled "Management Plan for the Review of Financial 
Management Education and Training". [Ref. 7:] This memorandum 
addressed a broad spectrum of financial management issues, and also 
spoke to the issue of quality control. Under the heading of Needs 
Assessment, curriculum configuration management and quality control 
was addressed as follows: 

Financial management course offerings fall generally into two 
categories: core courses that deal in basic concepts and 

principles that are independent of specific application . . . ; 
and service unique applications that train personnel to 
perform relatively narrow functions or apply basic principles 
in situations peculiar to a particular Service. 

In the case of core courses there is a need to ensure that the 

course content covers the subject matter at an appropriate level of 

comprehensiveness and sophistication. A second, related function 

is that of ensuring that what is taught is taught well. Whether 

analyzing new proposals or evaluating ongoing programs of 

instruction, assessments of teaching effectiveness are needed. 

[Ref. 7:] 

A survey was administered to collect detailed information on 
methods employed by the various agencies and departments to ensure 
the effectiveness of their courses and programs. The providers of 
financial management education and training courses and programs 
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were asked to complete a detailed questionnaire designed to assess 
their programmatic offerings. All providers of financial management 
education and training in DOD were surveyed. 

All military departments and defense agencies such as the 
Defense Logistics Agency and the Defense Finance and Accounting 
Service are included in the study. In-house as well as outside 
contracted providers were included, as were both civilian and 
military financial management education and training programs. 
However, financial management education and training programs based 
on the "correspondence" method were excluded from this research. 

In summary, this report attempts to answer the question of 
what methods are employed to assure the quality of financial 
management courses and programs within DoD. 

METHODOLOGY 

The methodology for conducting this research involved four 
distinct steps. Initially, the research concentrated on a thorough 
review of current literature, instructions, directives, reference 
materials and guidance dealing with financial management education 
and training. Information specifically concerned with financial 
management training course availability, course content, and 
targeted populations was reviewed. Applicable instructions and 
guidance for financial management personnel were also reviewed. 
Finally, the literature on quality assessment and assessment 
techniques was reviewed. 

Secondly, the development of a comprehensive survey instrument 
was undertaken. To help ensure a relevant and comprehensive survey 
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questionnaire was prepared, telephone interviews were conducted 
with administrators and professionals employed by providers of 
financial management education and training within DoD. To verify 
the contents and organization of the questionnaire, a pilot test 
was conducted with two providers of financial management education 
and training. The questionnaire was then distributed to all 
institutions within DoD providing financial management education 
and training. These institutions were identified by using the DoD 
Training and Performance Data Center listing of financial 
management education and training providers. The appendix lists 
agencies that responded to the survey. Finally, the survey 
responses were tabulated and analyzed, and in some cases telephone 
follow-up was conducted to clarify data from the responses. 

SCOPE 

A study conducted in 1990 by the Defense Manpower Data Center 
(DMDC) details the number of civilian and service personnel 
involved in financial/resource management. While the aggregate 
numbers presented in Table I below have been questioned by the 
military departments [Ref. 3: p.189] within DoD (e.g., the total 
acknowledged by these components is approximately 20,000 less than 
estimated by DMDC) , the numbers generally represent the total 
number of personnel engaged DoD-wide in the financial or resource 
management field. Table I details the findings of the DMDC 
study. [Ref. 3: pp. 187-188] 
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TABLE I, FINANCIAL/RESOPRCE MANAGEMENT EMPLOYEES OF POP 



CIVILIAN j MILITARY 



AGENCY 


PROF/ADMIN 


TECHNICAL 


OFFICER 


ENLISTED 


TOTAL 


ARMY 


27 , 105 


13,014 


2,746 


3,942 


46,807 


AF 


13,945 


5,647 


1,551 


5,376 


26,519 


NAVY 


15,513 


9,051 


688 


2,872 


28,124 


MC 


921 


1,194 


304 


1,314 


3,733 


DLA 


2,902 


2,671 






5,573 


OTHER 


8,124 


292 






8,416 


TOTAL 


68,510 


31,869 


5,289 


13,504 


119,172 



FINANCIAL MANAGEMENT EDUCATION AND TRAINING 



The task of training and educating the 100,000 or more 
personnel involved in financial management is a major enterprize. 
The training and education provided ranges from basic introductory 
courses in payroll, accounting and disbursing for junior enlisted 
personnel to graduate level Master's programs in financial 
management for military officers and mid-grade civilian employees. 

A total of 29 agencies and departments provide financial 
management and training. This number does not include the 
correspondence courses offered by the various military departments 
nor does it account for education obtained by military and civilian 
personnel outside of the structured programs recognized in this 
research. The diversity and number of courses offered are 
documented in Table II. [Ref. 3: p. 127] 
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TABLE II. DoD FINANCIAL MANAGEMENT COURSES BY PROVIDER 



Functional area: 


Army 


Navy 


Air Force 


DLA 


Total 


Budaet 


9 


2 




— 


13 


Finance 


37 


23 


19 


12 


91 


Analvsis 


27 


1 


19 


— 


47 


Resource Mcrmt 


27 


1 


— 


— 


28 


Totals 


100 


27 


40 


12 


179 



As Table II demonstrates, the 31 institutions offer 179 different 
courses of instruction. While firm data are not available, it is 
estimated that at least 10 percent of the financial management 
workforce, approximately, 10,000 personnel, attend financial 
management education and training programs and courses of some kind 
each year. [Ref. 3] 

II. SURVEY RESPONSES 

This section summarizes the findings obtained from the survey 
responses returned by financial management education and training 
providers, first, by general characteristics and then by question. 

A. General Characteristics of Financial Management Education 
and Training Courses and Programs 

Twenty-one of the thirty-one institutions surveyed responded 
within the time requirements specified in the survey instrument. Of 
the 21 responses to the survey 19 agencies provided detailed 
responses and two agencies replied that they did not conduct 
financial management education and training. 

The survey questionnaire was designed to gather basic data on 
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the characteristics of financial management education and training 
providers. Questions one through ten provided the respondents with 
the opportunity to describe their courses or programs in terms of 
target audience; number and length of those courses and programs; 
enrollment; average class size; and the level of instruction { e. 
g. , degree granting) . Several other questions also gathered 
general program information on these providers. For example, 
question 32 asked if the financial management program was supported 
by a library containing extensive DoD financial management related 
materials. The following analysis details the general 
characteristics of financial management education and training 
providers responding to the survey questionnaire. 

The respondents represent a wide spectrum of institutions that 
provide different types of financial management education and 
training. In terms of military sponsorship, five respondents were 
components within or sponsored by the Department of the Army, three 
within the Department of the Air Force, and eleven by the 
Department of the Navy (including one response from the Marine 
Corps) . 
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TABLE III 



SUMMARY OF GENERAL CHARACTERISTICS 



OF FINANCIAL MANAGEMENT EDUCATION AMD TRAINING PROVIDERS 



GENERAL CHARACTERISTICS 


NO. OF RESPONSES 


MILITARY SPONSORSHIP: 




ARMY 


5 


AIR FORCE 


3 


NAVY 


11 


LEVEL OF PROGRAM: 




ENTRY LEVEL 


1 


PROFESSIONAL 


9 


FULL SPECTRUM 


9 


AVERAGE ENROLLMENT: 




LESS THAN FIFTY 


2 


FIFTY TO ONE HUNDRED 


3 


ONE HUNDRED ONE TO TWO HUNDRED 


2 


MORE THAN TWO HUNDRED 


10 


NUMBER OF COURSES OFFERED: 




ONE 


5 


TWO TO FIVE 


4 


SIX TO TEN 


4 


MORE THAN TEN 


6 


TARGET AUDIENCE: 




MILITARY 


3 


CIVILIAN 


1 


BOTH 


15 


FACULTY COMPOSITION: 




MILITARY 


4 


CIVILIAN 


4 


BOTH 


11 



In addition to representing all of the military services, 
the respondents span the educational spectrum from those providing 
basic military entry level training to those awarding Master 
degrees. Nine of the nineteen respondents were classified as 
providing professional level education (e. g. , serving higher 
level civilian and military personnel or providing an advanced 
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educational degree. Of these nine, five offer graduate level 
programs leading to a master's degree. 

In terms of the size of the institutions responding to the 
survey, average annual enrollment in programs or courses ranged 
from a low of 36 to a high of 1,670. The total annual student 
population reported by the respondents was 7,324. Another measure 
of size is the number of courses offered. The range of courses 
offered by institution was from one to 36, with an average of 
eight. The target audience as reported by the survey respondents, 
not surprisingly, consists of both military and civilian employees 
of DoD. Only one agency reported its audience as civilian only, 
and three responded military only. Fifteen agencies serve both. 

In terms of faculty composition, e. g., military, 
civilian or a combination of both, four agencies used civilian 
instructors, four military and eleven had a mixture of military and 
civilian. 

B. Quality Assurance Methods Employed By Providers of 
Financial Management Education and Training 

The survey questionnaire was designed to elicit information on 
the quality assessment methods used by financial management 
education and training providers. Since the survey involved self- 
assessment of these methods, the responses provided represent only 
the views or opinions of the providers on quality assurance. 

No attempt was made to force the responses into a preconceived 
model of quality assurance. Since the survey is descriptive, the 
responses by providers of financial management education and 
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training represent their interpretation of what quality assurance 
consists of and the importance they attach to the methods used to 
assure the quality of their courses and programs. 

C. SURVEY RESPONSES BY QUESTION 

This section presents the responses to the survey questions 
organized by question. Those questions that required a "YES" or 
"NO" response are indicated by an asterisk preceding the question 
number. Each question is followed by a summary of the responses. 
Comments provided by survey respondents are also included. 

1. What is the target audience for vour program? 



The response option for this question was civilian, military 
or both. 



RESPON8E 


NUMBER 


PERCENTAGE 


CIVILIAN 


1 


5.3 % 


MILITARY 


3 


15.8 % 


BOTH 


15 


78.9 % 


TOTAL RESPONSES 


19 


100 % 



While the vast majority of respondents replied "BOTH", the 
majority of students are military. For example, several agencies 
replying "BOTH" reported military percentages in the 90 percentile 
range. The sole agency reporting only "CIVILIAN" was American 
University. American University offers a graduate level program 
for civilian employees of the Department of the Navy. 
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What are the approximate percentages of class attendees? 



The responses to this question varied, but as noted above the 
majority of students are military. Based on the 19 surveys 
received, approximately 80 % of all reported students are military. 

3. What are the ranges of grades or ranks of your attendees? 



GRADE/ RANK 


NUMBER 


PERCENTAGE 


ENTRY LEVEL 


1 


5.2 


"PROFESSIONAL" 


9 


47.4 


FULL SPECTRUM 


9 


47.4 


TOTAL 


19 


100.0 



Of the 19 responses received, nine provide financial 
management education for "professionals”. Professional in this 
analysis was defined as GS/GM 7's and above or military officers. 
Only one respondent provided services for junior enlisted or 
civilian personnel. The balance of providers offer financial 
management education across the full spectrum of civilian and 
military pay grades. 
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4 



How many separate courses in financial management are 
offered within vour r rnnr^m? 



COURSES 


HUMBER 


PERCENTAGE | 


1 


5 


26.3 


2-5 


4 


21.1 


5-10 


4 


21.1 


II > 10 


6 


31.5 



The range of courses offered as reported by survey respondents 
was from one to 36. The responses indicate that providers of 
financial management education and training have some difficulty 
defining what constitutes courses. This question was designed to 
identify the number of individual courses versus the number of 
programs offered. It is clear from the .survey data that some 
consistency is present in course definition. 

For example, based on the survey responses, the total number 
of courses reported was 163. Based on information previously 
collected this number was expected to be 181. Further analysis is 
required to understand the cause of this discrepancy. The most 
likely cause is the difficulty agencies have in identifying or 
categorizing specific courses as "financial management". 
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5. 



What is the length of vour program in days, weeks or 
months? 



LENGTH 


NUMBER 


PERCENTAGE 


5 DAYS OR LESS 


4 


23.5 


ONE MONTH 


2 


11.8 


1 MONTH - 1 YEAR 


4 


23.5 


ONE YEAR OR MORE 


7 


41.2 


TOTAL RESPONSES 


17 


100.0 



This question again points out the difficulty in the 
distinction between "courses" and "programs". While in some cases 
they may be the same, e. g. , the course and program constitute the 
only offering at an agency or department, the majority of survey 
respondents replied on the length of courses versus programs. 



6. How many times per year is it offered? 



FREQUENCY 


NUMBER 


PERCENTAGE 


ONCE 


3 


20.0 


TWO TO FIVE 


6 


40.0 


SIX TO TEN 


2 


13.3 


MORE THAN TEN 


4 


26.7 


TOTAL RESPONSES 


15 


100.0 
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What is the annual enrollment in vour program? 



ENROLLMENT 


NUMBER 


PERCENTAGE 


LESS THAN 50 


2 


11.8 


50 TO 100 


3 


17.6 


101 TO 200 


2 


11.8 


MORE THAN 200 


10 


58.8 


TOTAL RESPONSES 


17 


100.0 



The average (mean ) enrollment based on the seventeen 
responses was 431 students. The range of students was from 36 to 
1,670. The annual total student population reported by the 
responding agencies totalled 7,324. 

8. What is the average (mean) class size? 



SIZE 


NUMBER 


PERCENTAGE 


TEN OR LESS 


1 


5.3 


TEN TO TWENTY 


7 


36.8 


TWENTY TO THIRTY 


9 


47.4 


MORE THAN THIRTY 


2 


10.5 


TOTAL RESPONSES 


19 


100.0 



The average (mean) class size as reported by respondents was 
23.4. The range was from 9 to 60 students. 
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9. 



If vour program provides course credit for college, how 
many credits are offered in the total program? 



COLLEGE CREDIT GRANTED 


NUMBER 


PERCENT 


MASTER DEGREE 


5 


26.3 


CREDITS ONLY 


3 


15.8 


NO COLLEGE CREDIT 


11 


57.9 


TOTAL RESPONSES 


19 


100.0 



Eight responses were received on this question. Of the eight 
agencies responding, five were degree granting at the Master level 
with graduate credits ranging from 42 to 85. Three other agencies 
offered college credits of 6 hours, 16 hours, and from 3 to 16 
hours respectively. 

*10. Do students completing the program receive a degree or 
certificate? 



RESPONSE 


NUMBER 


PERCENTAGE | 


YES 


17 


10.5 1 


NO 


2 


J 

in 

• 

o 

H 


TOTAL RESPONSES 


19 


100.0 ll 



If ves. what is the title of this award? 

Of the 19 agencies responding to this question, ten issue 
certificates of completion, two issue certificates of training, one 
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issue a diploma and five issue masters degrees. Two agencies do 
not provide a degree or certificate to graduates. The total 
exceeds 19 due to one multiple response. 



11. How do vou determine need for new courses or instruction? 



The following comments were provided: 



• Coordination with the Defense Finance and Accounting Service; 
Coordination with the Comptroller of the Army or Assistant 
Secretary of the Army for Financial Management; Coordination 
with the Army's Training and Doctrine Command; and comments 
from the field. 

• Changes in the operating environment in DoD. 

• Feedback from participants; initiatives by faculty and 
directors of programs; guidance from the Assistant Secretary 
of the Army for Financial Management. 

• Assigned course proponents assess preliminary needs prior to 
an annual market survey which determines definitive needs. 

• New course start-up requirements are detailed in Army 
regulation 350-3 and are in the updated Training and Doctrine 
Command's Training Requirements Analysis System. 

• Inputs by users and field activities. 

• New course requirements are identified by the customers, 
usually through the use of Air Force form 19 - "Request to 
Establish New Continuing Education Short Course". Biannual 
curriculum reviews with DoD customers, and proceeding of 
functional boards can also identify needs. 

• Observation of changes in governmental and academic practices 
and problems. Consultation with the program sponsor. 

• Communication with the Navy's contract officer and adjunct 
professors who are professionals in the field. 

• New course and instructional needs are determined by on-going 
feedback from students, alumni, faculty and our educational 
service officers. 
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Needs survey of major claimants, the career planning board and 
participation in financial management professional 
associations . 

Fleet inputs. 

Student feedback plus input from the sponsor of the course. 

Input from the claimant; review of changes to official 

directives; input from field activities; input from class 

graduates; input from DoD; review of civilian sector 

approaches . 

Fleet needs; taskings from higher authority; critiques from 
other courses. 

Program changes; student and claimant feedback. 

Validated demands from field organizations for skills 
training. 

The normal cycle for determination of a new course of 

instruction begins with a change to operation in the 
community. The school will get input from headquarters that 
a task or course needs instruction. Initially, we get input 
from post-graduation surveys which indicate that new 
instruction is needed. Frequently, we distribute field 
surveys for more inputs. 



How do you determine the need to update or modify the 
instruction y° u currently offer? 



The following comments were provided: 



Changes in regulations or new directives; changes in computer 
programs related to financial management; lessons -learned, e. 
g. , Desert Storm; results of enlisted and officer development 
tests. 

Changes in regulations, directives or operating environment. 

Both courses have a two-dimensional evaluation program, 
subjective student end-of-course and instructor or facilitator 
evaluations. Courses are periodically reviewed by the course 
proponent, course manager and training quality assurance 



representative. Proponents provide input on regulatory and 
policy changes. 

• We conduct post-graduation surveys with graduates and their 
supervisors biannually to assess customer satisfaction. 

• Surveys, proponent inputs, MACOM inputs. 

• Input from field or functional managers; system changes; data 
provided from occupational surveys. 

• There are many possibilities. The faculty is responsible for 
maintaining currency in their field, as the field changes, so 
does the course. Other sources are end-of-course critiques, 
post-course critiques, curriculum reviews and communication 
with practitioners and sponsors. 

• Current issues in financial management plus guidance from 
senior military and DoD comptrollers. 

• Mid-course and end-of-course student evaluations. 

• In order to determine the need to update or modify the 
instruction currently offered, we test alternative delivery 
methods of instruction. 

• Interaction with students, instructional staff and policy 
makers from respective areas of subject matter. 

• Annual course reviews and periodic curriculum reviews 
conducted by fleet or type commander staff. 

• Student feedback, major claimant input along with changing 
issues in financial management. 

• End-of-course critiques and surveys of past graduates. 

• Course review; changes in reference material; changes in fleet 
needs . 

• Continuous review of instructors and student critiques. 

• The main determining factor for modifying our instruction is 
a change in the regulations governing financial management. 
Post-graduation feedback questionnaires and several forms of 
internal feedback also help to determine when updates need to 
be made. 
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*13 . Do vou have one or more sponsors or clients that directly 



influence your program of instruction? 



|| RESPONSE 


NUMBER 


PERCENTAGE 


YES 


18 


VO 

• 

-sj 


NO 


1 


5.3 


TOTAL RESPONSES 


19 


100.0 



If ves, who is/are the sponsor (s)? 

The following sponsors were specifically mentioned by survey 
respondents: 

« Secretary of the Air Force for Financial Management 

• Defense Finance and Accounting Service 

• Chief of Naval Operations 

• Navy Comptroller 

• Chief of Naval Education and Training 

• Navy Supply Systems Command 

• Fiscal Division, Headquarters, Marine Corps 

• CINCLANTFLT , COMNAVAIRLANT , CNO 

• Navy Bureau of Personnel 

• DoD Comptroller 

• Navy Bureau of Medicine 

• NAVFAC 

• Defense Acquisition University 

• NCD, NCB, CFMCP 

• Navy Exchange Command 
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Navy Food Service Systems Command 



*14 . Is course development guided or directed bv higher authority? 



RESPONSE 


NUMBER 


PERCENTAGE j 


YES 


13 


68.4 


NO 


6 


31.6 


TOTAL RESPONSES 


19 


100.0 



If so, who? 

• Soldier Support Center, CASCOM, TRADOC, ASA (FM) , DFAS 

• Assistant Secretary for the Army for Financial Management 

• Corp of Engineers Training Issues Committee 

• Headquarters, Air Training Command 

• OP-82/Director of Office of Budgets and Reports, NCB 

• NCD, NCB, NCF 

• Chief of Naval Technical Training 

• CNET and NAVFAC 
» BUMED 

• COMNAVAIRLANT 

• Standards Branch, Marine Corps 



15. What types of guidance do you receive? 



The following comments were provided: 



19 



• Regulations plus directed common core training. 

• Learning objectives. 

• Feedback from collective leadership and periodic assessment. 

• Proponents provide guidance on and approve the technical 
content . 

• Regulations and policy guidance. 

• Subjects of concern are the appropriate content for courses 
and the levels of learning required for each. 

• General guidelines as to curriculum content. 

• Subject matter expertise. 

• Selected topic material or requirements for graduate skill 
levels. 

• Course review, on-site evaluations; constant feedback is 
received from sponsors. 

• Curriculum reviews, directed input, issuance of new 
directives . 

• Formal instructions. 

• Internal curriculum review. 

• The Marine Corps uses the Instructional Systems Development 
process known as the Systems Approach to Training (SAT) as a 
guideline for all course development. 

16. How often do you receive such guidance? 

The typical response to this question was that guidance was 
received whenever necessary. Fourteen of the respondents answered 
in this fashion. Other responses were annually (1), quarterly (1) , 
and two to three times a year (1) . 
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*17 . Do you have staff dedicated to program development? 



[1 =“— 

RESPONSE 


NUMBER 


PERCENTAGE 


YES 


12 


66.7 


NO 


6 


33.3 


TOTAL RESPONSES 


18 


100.0 



If vea, hov many? 

The answers to this question varied from one to fifty-eight. 
The average response was between 1 and 3 . 



18 . What are the number of and percentages of civilian and/or 

military instructors in y° ur pr ogram? 

Of the sixteen agencies providing detailed information on this 
question, the following cumulative data were obtained: 

CIVILIAN: Number 142 Percentage 47 . 5 

MILITARY: Number 1§7 Percentage 52. 5 

The range of civilian employees is from 0 to 66; for military 
employees the range is 0 to 33. The average (mean) number of 
civilian employees is 6.3. The average (mean) number of military 
employees was 13.1. 
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What policies and criteria are used to assign or hire 
military facultv/instructors for vour program? 



The following responses were provided: 



• The officer or enlisted finance assignment branches screen and 
submit records of the best available personnel to the 
commandant who makes the final decision. 

• Subject matter expertise, previous schooling, previous 
assignments, and enlisted/officer fitness reports are all 
included as part of the review prior to actual assignment. 

« Grade, educational, experience and qualifications. 

• Military candidates are screened by a five member faculty 
committee. 

• A Master's degree is required. Broad knowledge of the 
comptroller field with broad background in at least one 
functional area. Strong performance record. 

• Must have graduate degree and are financial management 
specialists. 

• Must have subspecialty code? and 0-4 and above. 

• Manpower efficiency reviews conducted by Chief of Naval 
Education and Training. 

• Recent academic background and field experience. 

• NEC 9502. 

• New instructors are assigned on the basis of their technical 
knowledge of the subject matter they will teach and their 
communication skills. 



20 . What policies and criteria are used to appoint or hire 

civilian faculty/instructors for vour program? 



The following comments were provided: 
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• All civil service positions within the training instructor 
field have prescribed minimum qualifications. 

• College degree, subject matter expertise, ability to teach. 

• Coordination/consultation with department chairs and academic 
directors. 

• Instructors are recommended by their supervisors. 

• Review of job series and job experience. Depending on grade, 
career field experience is generally required. Final 
selection is based on educational requirements, subject matter 
knowledge, experience, interview, and potential to perform on 
the "platform". 

• Grade, educational experience, qualifications and OPM 
guidance. 

• Civilian candidates are screened by a five member faculty 
committee. 

• Must have doctoral degree. For senior faculty, must also have 
scholarly research record. 

• Resumes, references, interviews. 

• Level of degree, professional field experience in finance. 



*21. Are professional qualifications standards required for 
facultv/instructors (e.q.. degrees. prior experience, 
completion of an instructor training program)? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


17 


94.4 


NO 


1 


5.6 


TOTAL RESPONSES 


18 


100.0 



If ves, please describe. 



• College degree. 
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• Same standards as for the University. 

• Most instructors have a degree in their subject area (minimum 
of bachelor's) and not less than five years experience in 
their subject area. Further, instructors must complete the 
Instructional Methods Course. 

• All military officers have degrees; civilians usually have 
degrees. The Faculty Development Course must be completed 
within 90 days of assignment. 

• Associates degree or higher? completion of Technical Training 
Instructor Course. 

• All faculty must have an appropriate Master's degree and 
experience in the field taught or a closely related field. 
Faculty hired without significant prior teaching experience 
must complete Academic Instructor School. 

« Master's degree plus completion of Air Force Academic 
Instructor School. 

• We prefer faculty with Phd's and teaching experience. 

• Each faculty member in the teaching discipline at the master's 
degree level will hold the terminal degree in the teaching 
discipline or a related discipline. Outstanding professional 
experience plus a Master's degree is preferred. 

• Specific Navy enlisted classification codes apply to some 
billets which require fleet experienced personnel to be 
detailed to the billet. 

• Minimum of Bachelor, desired Master's and five years of 
experience. 

• Instructor Training School? subspecialty designation. 

• Graduate of instructor training school. 

• Prior experience in the field. 

» All instructors complete a two week Instructor Training 
Course. 
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*22 . Do you use classroom observation to evaluate instruction? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


17 


89.5 


NO 


2 


10.5 


TOTAL RESPONSES 


19 


100.0 



If ve3. who performs this observation? 

The following responses were provided: 

• Department directors, division chiefs, course directors, 
commandant . 

• Staff and faculty development representatives. 

• Peers. 

• Program directors. 

• Classroom courses are evaluated by the course manager, quality 
assurance representative, proponent or lead instructor. 

• School deans, department chairman, course directors. 

• Instructor supervisors and/or flight commander. 

• Department head. 

• Commandant . 

• Regional academic director. 

• Educational specialists. 

• Staff instructors. 

• Academic Standards department. 

• Academic Program division. 

• Qualified instructors. 

• Training director and department head. 
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• Master course manager. 

• Curriculum Standards Branch Officer, Instructional Operations 
Officer, Academic Officer, Academic Chief, Instructional 
Operations Chief. 



How often is it performed? 



The following responses were provided: 

• Weekly - 5 responses 

• Quarterly - 2 responses 

• Twice yearly - 4 responses 

• Annually - 6 responses 

• Unscheduled or as appropriate - 2 responses 
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*23 . Are other methods used to evaluate classroom instruction? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


15 


83.3 


NO 


3 


16.7 


TOTAL RESPONSES 


18 


100.0 



If ves. Please describe. 



Of the fifteen agencies which responded to this question, all 
cited student critiques as a method used to evaluate classroom 
instruction. Detailed responses are listed below: 



• Student evaluation of training. 

• End-of-class evaluation by students. 

• Surveys from participants; group assessment sessions. 

• End-of-course questionnaires by students ; audits of classes by 
proponent representatives; post-graduation surveys sent to 
graduates and their supervisors 6 months after course 
completion; academic excellence analysis. 

• ATC form 736; student critiques; Training evaluation Reports; 
Training Quality Reports from graduates and supervisors of 
graduates; field visits; STAN evaluations and Inspector 
General inspections. 

• We use student/supervisor post course critiques. The student 
class leader debriefs the department head at the conclusion of 
each course. 

• Student opinion questionnaires. 

• Mid-course and end-of-course student evaluations. 

• Student feedback instruments; discussions with 
instructors/administrators . 

• Student feedback from every class. Representatives from 



27 



sponsor also monitor classes. 

• Student critiques; peer review. 

• Course reviews; student critiques; external evaluations 
(surveys) . 

• Quarterly improvement form completed by all students. 

• Course evaluation upon student course completion and a 
supervisor's evaluation of student performance in training 
skills 3 months after course completion. 

• Every instructor completes an after instruction report to note 
any problems with lesson materials. Each student fills out an 
end-of-course critique prior to graduation which enables him 
to evaluate the overall course material. 



*24 . Are other methods used to evaluate instructors? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


12 


63.2 


NO 


7 


36.8 


[| TOTAL RESPONSES 


18 


100.0 || 



If Y es * Please describe. 

The following responses were received: 

• Selection of instructor of the month , • quarter and year. 

• Survey of a sample of students and their supervisors conducted 
six months after course completion to determine adequacy and 
application made of training. 

• End-of-course questionnaires; performance appraisals; informal 
visits to classroom by course directors. 

• Student critiques; STAN evaluations; Inspector General 
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inspections. 

• The student course critique covers the instructor's 
performance. 

• Student opinion questionnaires. 

• Faculty may choose a 15 minute tape presentation, arrange a 
supervisory visit and evaluation or submit a representative 
portfolio for evaluation purposes. 

• Student feedback instruments. Follow-up discussions with 

students/administrators . 

• Student critiques; peer review. 

• Course review; student critiques; external evaluations. 

• Review of instructor prepared course material . 

• 100 percent of the students of each class complete an 
Instructor Rating form. This form allows them to evaluate the 
lesson material as well as the effectiveness of the 
instructor. 



*25 . Do you employ productivity measures to evaluate 
instructors? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


3 


15.8 


NO 


16 


84.2 


TOTAL RESPONSES 


19 


100.0 



If so. please describe. 

Only three of the nineteen respondents indicated that 
productivity measures were employed. Of these three only one 
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respondent provided measures employed. These were: 
instructor contact hours; student training years per instructor; 
instructor contact hours per student training years produced; and 
consulting hours. 



*2 6. Is your institution accredited? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


9 


47.4 


NO 


10 


52.6 


TOTAL RESPONSES 


19 


100.0 



If ves. bv whom? 



• North Central Association of Colleges and Universities, 
Commission on Institutions of Higher Education. 

• AACSB for Schools of Management. 

• Southern Association of Colleges and Schools (2 responses) . 

• North Central Association. 

• American Council on Education (2 responses) . 

• Western Association of Schools and Colleges. 

• Middle States Association. 

How often? 

Frequency varied from annually to every ten years. 

Typically, the review cycle is five years. 
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*27. Is vour institution reviewed by other agencies? 



1 RESPONSE 


NUMBER 


PERCENTAGE i 


f 

YES 


13 


68.4 


NO 


6 


31.6 


TOTAL RESPONSES 


19 


100.0 



If ves. by whom and how often? 



Responses received included: 



• Army Training and Doctrine Command. 

• American Council on Education (2 responses) . 

• Defense Finance and Accounting Agency; Secretary of the Air 
Force for Financial Management; Air Force Audit Agency; 
Community College of the Air Force. 

• Air University (2 responses) . 

• National Association of Schools of Public Affairs and 
Administration . 

• MIVER and QES . 

• Commander, Training Command, U. S. Atlantic Fleet. 

• CNET; CNTECTRA; NAVFAC . 

• Navy and Bureau of Medicine Inspector's General. 

• Type commander. 

• Command inspections. 
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What ia the nature and extent of this review? 



The following responses were provided: 



• Standard ACE review of course material and lesson plans. 

• Courses and library support are evaluated by ACE teams of 
subject matter experts. Evaluators consider factors such as 
course objectives, subject matter, level of difficulty, 
duration, methods of student evaluation, applicability to 
post-secondary education programs, and background and 
selection of students and instructors. ACE then makes 
recommendations to colleges/universities for granting academic 
credit. 

• To determine accuracy and currency of training. 

• Primarily focuses on management. 

• Curriculum review. 

• Year-long self-assessment followed by site visit. 

• Formal Inspector General inspections. 

• Review of curriculum and adherence to directives. 

• Inspections. 
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What methods or modes of instruction are utilized in vour 



prftfTfwm (e.q.. lecture, discussion# simulation, 
independent study}? 



MODE OF INSTRUCTION 


NUMBER 


PERCENT 


LECTURE 


16 


94.1 


DISCUSSION 


10 


58.8 


SIMULATION 


8 


47.1 


CASE STUDY 


7 


41.2 


SEMINARS 


2 


11.8 


INDEPENDENT STUDY 


6 


35.3 


PROJECT TEAMS 


3 


17.6 


FIELD TRIPS 


1 


5.9 


LABORATORIES 


2 


lnng?p H«8 


PERFORMANCED BASED 


2 


11.8 



The typical responses to this question included lectures, 
discussions, case studies, simulation and self or independent 
study. 



30. What percentage of instruction Is delivered in each mode 

you identified? 



Of the eleven institutions providing detailed percentages on 
modes of instruction, all cited lectures and discussions as the 
predominant modes utilized. On average, lectures represented 40 
percent of the instructional program; discussion represented 20 
percent. Simulation, case study and independent study represent 
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approximately 10 percent each. The remaining 10 percent was 
divided among the modes reported in Question 29 above. 

31. What types of course reading materials are used in vour 

p-rnqrwm (e.q. . textbooks. published articles. DoD 
official documents, self-generated materials)? 

Sixteen agencies responded to this question. Thirteen cited 
DoD official documents. Ten developed their own (self-generated) 
materials. And nine utilized textbooks. 

*32 . Is vour program supported bv a library containing extensive 
DoD financial management related 
materials? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


9 


50.0 


NO 


9 


50.0 


TOTAL RESPONSES 


18 


100.0 



Of the eighteen responses received, half noted that they were 
not adequately supported by a library containing extensive DoD 
financially related materials. Question 33 and 34 asked 
specifically about library support. Six providers specified 
improvements that would like to see in library support (See 
Question 33) and six noted that improvements were planned in 
library support (See Question 34) . 
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33 . How would you like to improve library support for vour 

program? 

Six responses were provided on this question. They included: 

• Establish a library within our school. 

• Be able to interact with some DoD instructional material data 
base. 

• Ensure that the library has all required materials. 

• Improve library support by joining the Washington area 
consortium. 

• Specified funds for financial texts. 

• Obtain sufficient copies of reference materials. 



*34 . Are there improvements in vour library services planned? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


6 


35.3 


NO 


11 


64.7 


TOTAL RESPONSES 


17 


100.0 



If ves. please explain these improvements and 
Indicate when they will occur. 



• An effort is underway to automate the Logistics Library which 
will provide computer assisted in-house services plus remote 
dial-in access. (Scheduled for FY 1993) 

• Our library has just assigned collection development 
specialists to enhance library support for school programs. 



35 



*35. Is your pi’oqT'am supported bv a computer center or 



laboratory? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


14 


73.7 


NO 


5 


26.3 


TOTAL RESPONSES 


19 


100.0 



If no. what computer resources are employed at vour 
institution. 



Only one response was provided in response to this portion of 
the question; that response indicated that only personal computers 
were used. 

*36. Does vour instructional program include computer 
exercises or assignments? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


16 


CD 

CD 

• 

VO 


NO 


2 


11.1 


TOTAL RESPONSES 


18 


100.0 1 



If yes, please describe how computers are utilized. 
Responses included: 

• Standard Army Information Management System. 

• Real-world applications. 
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• Familiarize clients with computers. 

• Simulation. 

• Data retrieval exercises; National Budget case. 

• Self-paced instruction. 

• Application practices. 

*37 . Is vour program wai 1 supported with respect to 



adnrl tH strative staff, funding for quest 
speakers. etc.? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


12 


63.2 


NO 


7 


36.8 


TOTAL RESPONSES 


19 


100.0 



What improvements in support would be desirable? 

Of the four agencies that specified improvements, all noted 
additional funding for guest speakers would be desirable. 

38 . What methods are used to determine the need for vour 

current program? 



The following responses were provided; 



• The Army conducts annual surveys of all users of Army training 
programs . 

• Sponsors. 

• Needs assessments are conducted by proponents prior to any 
consideration of course design or development. Proponents 
annually verify the need for classroom courses and an annual 
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market survey provided the needs verification from the users. 

• The Total Army Centralized Individual Training Solicitation 
(TACITS) is a survey which is conducted annually for primary 
information and follow-up; Structure Manning Decision Review 
is conducted annually; the Training Resources Arbitration 
Panel (TRAP) process is done monthly; off-line memorandums can 
be submitted once the TRAP closes for execution; curriculum 
council reviews; annual course proponent reviews. 

• Field evaluations; Training Evaluation reports; Training 
Review Analyses; occupational surveys. 

• Curriculum review addresses this specific issue. The need is 
also assessed by the quantity of requests for any particular 
course. 

• The need for resource management education among military 
services. 

• Data on P-coded billets and other financial management 
positions. 

• Program reviews are conducted every five years. 

• Alumni surveys; student surveys; end-of-course assessments; 
ESO inputs; and faculty surveys. 

• NC and NCD guidance; Career Program Planning Board; 
initiatives in the operational environment. 

• Site surveys conducted by systems command sponsors and annual 
course curriculum reviews. 

• Student feedback and sponsors' requirements. 

• Demand for student placements. 

• Fleet need; tasking from higher authority; student critiques. 

• Review by claimant. 

« Extensive course review by program manager and field 
personnel. 

• The Marine Corps uses the Individual Training Standards that 
describe the specific tasks and knowledge required. We also 
conduct Course Content Review Boards to ensure the training 
standards are correct. 
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39 . What methods are used to determine the validity. 

accuracy, relevance and timeliness of vour current 

program? 



The following methods were cited: 



• Annual review of training strategy by subject matter experts; 
annual review of lesson plans; needs analysis. 

• Proposed training program based on TACITS data is reviewed and 
approved by the Department of the Army. 

• Accreditation and standards of graduate program. 

• Proponents are charged with the responsibility for technical 
content. Proponent/ instructor meetings as well as monitoring 
of courses are performed to ensure technical accuracy, 
timeliness, etc. Student end-of-course critiques are 
constantly monitored for student perceived requirements for 
change. 

• Curriculum council reviews; annual course proponent reviews; 
end-of-course questionnaires; Structure Manning Decision 
review; Army Training Resource Requirements System. 

• Field evaluations; Training Evaluation Reports; Training 
Review Analysis; occupational surveys; Inspector General 
inspections; student critiques. Also a customer service 
information line has been installed. 

• Curriculum reviews; visits with the sponsor; student 
critiques; functional boards; and other feedback. 

» Review by senior financial management officials; student 
feedback. 

• Observation of changes in governmental and academic practices 
and problems. Consultation with program sponsor. 

• Surveys and interviews with students, alumni and practitioners 
in the field. 

• All programs are under continuous review. Inputs from the 
teaching site is gathered each term to determine the validity 
of our current program. 

• Subject matter experts; career board annual report, classroom 
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visits, surveys of the community. 

• Annual course reviews; systems command assessments and 
technical audits. 

« Sponsor review; willingness of users to reimburse for the 
course . 

• Curriculum survey to past graduates; review by sponsors. 

• Course reviews; standardization conferences. 

• Student/claimant feedback. 

« Competent job performers and subject matter experts review 
course content for validity, accuracy and relevance. 

• Following completion of each course, a career content review 
board is conducted to ensure the material is accurate, timely 
and relevant. 

What methods are employed to determine student competence 
during and upon pro gram completion (e.q., passing 
standardized tests, written course work, observation of 
performance on the -job)? 

Responses provided included; 

• Practical exercises (4 responses) 

• Tests (14 responses) 

• Research papers (4 responses) 

• Individual or group projects (3 responses) 

• Case studies (4 responses) 

• Oral presentations (3 responses) 

• Passing national exams (2 responses) 

• Thesis (1 response) 
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*41. Do you have a formal method of student course evaluation? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


19 


100.0 


NO 


0 


0.0 


TOTAL RESPONSES 


19 


100.0 



If ves. describe this process. 

Of the nineteen responses received to this question, all 
indicated that a formal method of student course evaluation 
existed. All used a student critique form for this purpose. 

*42 . Are other methods used to permit students to 
evaluate vour instructional program? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


10 


52.6 


NO 


9 


47.4 


TOTAL RESPONSES 


19 


100.0 



If ves. describe these methods. 

The following comments were provided: 

• The Commandant holds periodic informal discussions with 
students. 

• Post-graduation surveys (3 responses) . 

• DD form 1556 is completed by students and their supervisors. 
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• Student-to-instructor feedback. 

• Students provide informal comments to their class leaders, who 
can relay them at end-of-course debriefs. 

• Students are invited to meet with the department chairman 
periodically and can make individual comments to him as they 
wish. 

• Follow-up with educational specialists. 



*43. Are vour students required to pass anv standardized 
DoD or natio nal <nr»m inations (e.a., CPA. CMA) ? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


1 


5.3 


NO 


18 


94.7 


TOTAL RESPONSES 


19 


100.0 



The one respondent answering "yes" to this indicated 
that students are required to pass the GRE, GMAT or MAT. 



44 . What percentage of students entering vour program 

successfully complete it? 



Of the eleven responses to this question, 10 indicated that 
completion rates averaged over 95 percent. One agency cited a 
completion rate of 75 percent. 
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*45. Do you track the performance of vour graduates? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


7 


36.8 


NO 


12 


63.2 


TOTAL RESPONSES 


19 


100.0 



If ves. hov is this done? 



Of the seven responses provided, six agencies cited the use of 
post-graduate surveys as the principle method utilized to track 
graduate performance. Specific comments were: 



• School liaison teams visit finance units in the field to 
evaluate school training and training needs with graduates and 
their supervisors. Survey of a sample of students and their 
supervisors is conducted about six months after course 
completion to determine adequacy and applicability of 
training. 

• Post graduate surveys are sent to graduates and their 
supervisors six months after course completion. 

• Field visits by training evaluators. 

• We use one year and five year out surveys of graduates. 

• Follow-up questionnaire one year after completion; survey of 
supervisors of graduates of our program; track progression 
through the alumni association. 

• A post-course evaluation is sent to the student's supervisor. 
It is intended to determine the effectiveness of training more 
than the performance of the individual. 

• The supervisors of all graduates are sent a feedback 
questionnaire to help us evaluate if the student learned the 
necessary prerequisite knowledge. 
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*46. Are formal methods employed to solicit feedback from former 



students and clients about the utility of vour pr ogram? 



RESPONSE 


NUMBER 


PERCENTAGE 


YES 


12 


63.2 


NO 


7 


36.8 


TOTAL RESPONSES 


19 


100.0 



If ves. how is this done? 



Of the twelve responses received, nine agencies cited the use 
of post-graduate surveys for this purpose. Post-graduate surveys 
were used for two purposes: (1) To track the performance of 

graduates (see Question 45 above) , and (2) To assess the 
effectiveness or utility of the program. 

47 . In vour opinion, what are the most important methods you 

employ to ensure the quality of vour program? 

This question was designed to be the cap-stone question of the 
survey. It provided the opportunity to summarize the methods used 
by survey respondents to ensure quality. Responses included: 

• Visits to finance units in the field? selection of 
instructors/subject matter experts who have field experience; 
insuring that we are kept informed on the numerous changes in 
finance, accounting and resource management regulations and 
policies. 
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• Qualified faculty; feedback from students and their 
supervisors . 

• Recognition by sponsors and commands/agencies which provide 
participants for the program; competition for individuals to 
attend; long standing reputation of the program. 

• Needs assessment; task analysis; analysis of pre/post test 
results; quality assurance; proponent and course manager 
evaluations; periodic proponent/ instructor/SME workshops for 
technical accuracy and relevance; analysis of student end-of- 
course evaluations. 

• Close attention to post-graduation surveys; Close attention to 
comments on end-of-course questionnaires; continuing dialogue 
with students and others in the functional area; exceptional 
instruction; extensive knowledge of subject matter and 
regulatory requirements. 

• Field visits; student evaluations; rotation of military 
instructors; field feedback; workshops to review training. 

• We attempt to hire and maintain a top quality faculty. The 
Faculty Academics Standards Committee reviews each course 
every three years. 

• Constant review of the curriculum by the faculty plus biannual 
review by Air University and senior DoD comptrollers; student 
feedback. 

• Faculty involvement with the Navy and Defense Department and 
in academic activities and organizations. 

• We set high standards for professional graduate level work and 
expect the faculty to meet these standards. 

• Student course assessments; faculty observations. 

• Review of material for accuracy of content; student and 
instructor evaluations; post-training job performance. 

• Periodic fleet/type commander staff reviews and systems 
command technical assessments. 

• Constant feedback from the students and inputs from the course 
sponsor. 

• Collection of data via student critiques; trend analysis over 
time. 

• Fleet inputs. 
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• On-going student and client feedback. 

• Feedback from students and their supervisors. 

• The Instructional Systems Development Process guides us to 
ensure the quality of our program. Additionally, our school 
enjoys an outstanding relationship with field personnel. We 
are able to respond to changes to regulations and provide 
better instruction. 



48. Is there anv other information that you would offer on 

quality assessment at vour institution that we did not 
request? 

No responses were provided to this question. 



III. ANALYSIS OF SURVEY RESPONSES 
Analysis of responses begins with the broadest element of the 
survey — responses to question 47. Question 47 asked — What are the 
most important methods you employ to ensure the quality of your 
program? The question gave respondents the opportunity to summarize 
the methods they used to assure quality. Responses to this 
question form the basis for evaluating the quality assurance 
methods employed by financial management education and training 
providers. 

Quality Assurance Methods Employed 

The following list summarizes methods cited by survey 
respondents to ensure the quality of their programs: 

• Student feedback (11 responses) . 
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• Sponsor guidance/ feedback/support (6 responses) . 

• Hiring/ensuring quality faculty (6 responses). 

• Feedback/contact with clients (9 responses) . 

• Tracking the post-graduation performance of students (5 

responses) . 

• Monitoring of faculty performance (3 responses). 

• Keeping abreast of changes in financial management (4 

responses) . 

• Tracking student performance in courses and/or programs (2 
responses) . 

The methods noted by financial management education and 
training providers in their responses to this survey question 
(Question 47) also are supported by the responses made to other 
questions within the survey questionnaire. Each of the methods 
cited above are discussed separately below with references to other 
survey questions as appropriate. 

1. Student Feedback 

The use of student feedback, specifically end-of-course 
critique forms, is a response that appears throughout the surveys 
returned by financial management education and training providers. 
In addition to Question 41 which asks — Do you have a formal 
method of student course evaluation? to which all nineteen 
respondents affirmatively replied, the questions and the number of 
respondents that listed student feedback as a prime determinant are 
as follows: 

• Question 11: How do you determine the need for new courses or 

instruction? (6 responses) 
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• Question 12: How do you determine the need to update or 

modify the instruction you currently offer? (10 responses) 

• Question 23: Are other methods used to evaluate classroom 

instruction? [Note: Question 23 asks whether respondents use 
classroom observation to evaluate instruction] (15 responses) 

• Question 24: Are other methods used to evaluate instructors? 

(7 responses) 

• Question 38: What methods are used to determine the need for 

your current program? [Follow-on to questions 11 and 12] (3 

responses) 

• Question 39: What methods are used to determine the validity , 

accuracy, relevance and timeliness of your current program? 
(7 responses) 



2 . Sponsor Guidance 

Sponsor guidance is another common method of quality 
assurance employed by respondents to the survey. Question 13 (Do 
you have one or more sponsors or clients that directly influence 
your program of instruction?) directly measures this practice. 
Eighteen of the nineteen respondents acknowledged sponsor guidance. 
Question 14 (Is course development guided or directed by higher 
authority?) also addresses this practice. Thirteen of nineteen 
respondents responded "yes" to the question. 

Additionally, the following survey questions applied to 
the sponsor guidance [ the number of positive responses is shown] 

• Question 11: How do you determine need for new courses or 

instruction? (9 responses) 

• Question 12: How do you determine the need to update or 

modify the instruction you currently offer? (8 responses) 

• Question 27: Is your institution reviewed by other agencies 

[other than accreditation]? (7 responses) v. 
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• Question 38: What methods are used to determine the need for 

your current program? (6 responses) 

• Question 39: What methods are used to determine the validity, 

accuracy, relevance and timeliness of your program? (8 
responses) 

3. Hiring Quality Faculty 

Hiring of quality instructors or faculty was the third 
quality assurance method cited by survey respondents. In this 
regard, the issue of faculty ’’quality" or competence was addressed 
directly by the survey in questions 19 through 25. Question 21 is 
the most specific. It asked - Are professional qualification 
standards required for faculty/instructors? Seventeen of 18 
respondents stated that they employed this method. 

Also relevant for the discussion of quality faculty were 
questions 19 and 20. The questions asked - What policies and 
criteria are used to hire military (Question 19) or civilian 
(Question 20) faculty/ instructors for your program. Survey 
responses to both questions focused on experience, education and 
other factors that prepare individuals for the teaching profession. 
In fact all respondents noted that they used selected criteria to 
ensure the hiring of qualified individuals. 

Question 22 through 25 further addressed the issue of 
instructor or instruction observation to promote quality faculty. 
The issue of faculty on-the-job performance is discussed in 
subsection (f) Faculty Performance. 
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4. Feedback from Clients 

Client feedback was the fourth method cited by survey 
respondents to ensure quality. Question 13 addressed this method 
by asking — Do you have one or more sponsors or clients that 
directly influence your program of instruction? As the question 
implies, the distinction made between clients and sponsors was not 
always clear in the responses. Only one respondent specifically 
mentioned a client that influenced the program. Nevertheless, 
other survey question responses noted the importance that financial 
management education and training providers give to client 
feedback. 

The following questions applied: 

• Question 11: How do you determine the need for new courses or 

instruction? (9 responses) 

• Question 12: How do you determine the need to update or 

modify the instruction you currently offer? (3 responses) 

• Question 38: What methods are used to determine the need for 

your current program? (2 responses) 

• Question 39: What methods are used to determine the validity, 

accuracy, relevance and timeliness of your current program? 
(3 responses) 

Additionally, client feedback is obtained through the use 
of post-graduation surveys to the extent that graduates and their 
supervisors represent the clients of the courses and programs 
offered. This practice is more fully examined below. 

5. Post-graduation Performance of Students 

The post-graduation performance of financial management 
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education and training students was the fifth method of quality 
assurance indicated by survey respondents. This method was 
specifically addressed in the survey in Question 45. The question 
stated — Do you track the performance of your graduates? Seven of 
the nineteen respondents did utilize such tracking. Of these 
seven, six used post-graduation surveys for this purpose. 

6. Faculty Performance 

A sixth method of quality assurance reported was faculty 
performance. This method is similar to the hiring of quality 
faculty previously discussed in subsection (c) . However, the 
monitoring of faculty performance is an on-going process that, in 
principle, verifies the hiring policies and procedures of financial 
management providers. 

Questions 22 through 25 addressed the issue of faculty 
performance. These questions with the number of affirmative 
responses are as follows: 

• Question 22: Do you use classroom observation to evaluate 

instruction? (17 responses) 

• Question 23: Are other methods used to evaluate classroom 

instruction? (15 responses) 

• Question 24: Are other methods used to evaluate instructors? 

(12 responses) 

• Question 25: Do you employ productivify measures to evaluate 

instructors? (3 responses) 

One of the prime measurements of faculty performance was 
the use of student critiques (end-of-course evaluations) and 
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student feedback. For example, all fifteen of the responses to 
question 23 cited the use of student critiques as the method used 
to evaluate classroom instruction. Similarly, eight respondents to 
question 24 noted student critiques as the primary method employed 
to measure instructor performance. 

7. Keeping Abreast of Changes 

Keeping abreast of changes in financial management was 
the sixth quality assurance method cited by survey respondents. 
Keeping abreast of changes in financial management was noted in the 
following survey responses by question: 

• Question 12 : How do you determine the need to update or 

modify the instruction you currently offer? (5 responses) 

• Question 39: What methods are used to determine the validity, 

accuracy, relevance and timeliness of your current program? 
(2 responses) 

8. Tracking Student Performance 

The final method reported to ensure quality was the 
tracking or monitoring of student performance while attending the 
financial management institution. The survey addressed the issue of 
student performance or competence in questions 40, 43 and 44. 

Question 40 asked — What methods are employed to 
determine student competence during and upon program completion (e. 
g. , passing standardized tests, written course work, observation of 
performance on the job)? Question 43 asked about the requirement 
for students to pass national examinations. Only one respondent 
required this. Finally, Question 45 asked respondents what 
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percentage of students entering the program successfully complete 
it? The average successful completion rate was in excess of 95 
percent. 

Additionally, while this method is directed toward in- 
house success, post-graduation performance also informs the 
institution on the quality of their program. 

SUMMARY 

The preceding sections analyzed responses to the survey of 
financial management education and training quality assurance 
methodology. Table IV summarizes the methods used. The percentage 
column is calculated on the basis of the number of responses 
divided by the total number of responses received. 



TABLE IV. SUMMARY OF QUALITY ASSURANCE METHODS 



Method 


Responses 


Percentage 


(a) 


Student Feedback 


11 


57.9 


(b) 


Sponsor Guidance 


6 


31.6 


(c) 


Hiring Quality Faculty 


6 


31.6 


(d) 


Feedback from Clients 


9 


47.4 


(e) 


Post-graduation Performance 


5 


23.3 


(f) 


Faculty Performance 


3 


15.8 


(g) 


Keeping Abreast of Changes 


4 


21.1 


(h) 


Tracking Student Performance 


2 


10.5 



53 



IV. CONCLUSIONS 



The respondents to this study represent a wide spectrum of DoD 
institutions currently providing financial management education and 
training. Five institutions responding operate within or are 
sponsored by the Department of the Army, three by the Department of 
the Air Force, and eleven by the Department of the Navy. 

The respondents span the educational spectrum from those 
granting Master degrees to those providing basic military entry 
level training. In terms of the size of the institutions 
responding to the survey, average enrollment in programs or courses 
ranges from a low of 36 to a high of 1,670. The range of courses 
offered is from one to 36, with an average of eight. 

The target audience as reported by the survey respondents was 
both military and civilian employees of DoD. Only one agency 
reported its audience as civilian only, and three responded 
military only. The remainder serve both. 

In terms of faculty composition ( military, civilian or a 
combination) , four agencies use civilian instructors, four 
military, and eight have a mixture of military and civilian. 

The question addressed by this research was the method of 
quality assurance employed by financial management providers. The 
responses to the survey of quality assurance methods are listed 
below. 

(a) Student Feedback 

(b) Sponsor Guidance 
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(c) Hiring Quality Faculty 

(d) Feedback from Clients 

(e) Post-graduation Performance 

(f) Faculty Performance 

(g) Keeping Abreast of Changes in DoD Financial 

Management 

(h) Tracking Student Performance 

Student feedback and feedback from clients were the two 

quality assurance methods most commonly cited by survey 
respondents. The survey results, however, clearly point out the 
wide divergence of quality methods used by financial management 
instruction providers. 

In addition to the methods cited by survey respondents to 
ensure quality, other quality measurement methods might be 
employed. Accreditation and assessment of the quality of support 
resources also are quality assurance approaches that could be 
employed by financial management education and training providers. 

(1) Accreditation 

An accreditation process provides an independent 
evaluation of the performance of educational institutions. 
Accreditation organizations typically review courses and programs 
for content, instructor competence, and the availability of 
ancillary services including libraries and computers. Question 26 
of the survey asked - Is your institution accredited? Nine of the 
nineteen respondents replied affirmatively. The accrediting 
organizations mentioned included the North Central Association of 



55 



Colleges and Universities, Commission on Institutions of Higher 
Management, the American Council on Education and the Western 
Association of Schools and Colleges. Existing accreditation 
processes are not applicable to all of the respondents to the 
survey. However, accreditation would appear useful to ensure the 
quality of all DoD courses and program offerings, if such a process 
accommodated the unique nature of DoD Financial Management 
education and training institutions. 

(2) Evaluating the Adequacy of Institutional Support 

Another method for ensuring quality is to assess the 
adequacy of institutional support for instructional courses or 
programs . 

The survey questionnaire asked several questions on 
institutional support: 

• Question 32: Is your institution supported by a library 

containing extensive DoD financial management related 
materials? 

• Question 35: Is your program supported by a computer center 

or laboratory? 

• Question 37: Is your program well supported with respect to 

administrative staff, funding for guest speakers, etc.? 

Responses to these question were mixed. Fifty percent of the 
respondents reported that their program was not supported by a 
library with extensive DoD material. Seven of nineteen respondents 
answered that their program was not well supported with respect to 
administrative staff or funding for guest speakers. Four responses 
identified funding for guest speakers as the only deficiency in 
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this area. Adequate support is necessary to assure quality of 
instruction. 

While it might seem obvious that developing and maintaining 
quality programs necessitates the involvement not only of the 
provider but also of the sponsor and clients, the overall response 
rate indicates that this is not always the case. Furthermore, the 
differences in responses on this issue suggests that more effort 
needs to be expended toward the development of methods that ensure 
financial management education and training is relevant to both 
sponsors and clients. 

V. RECOMMENDATIONS 

There is no single "best method" to assure quality in DoD 
financial management education and training. A mix of different 
approaches appropriate to the mission and nature of instruction 
provided at each institution is needed. Responses to the survey of 
DoD FM instruction providers indicate that a number of approaches 
to quality assurance are employed currently. However, there is 
room for improvement. Based upon the findings from this survey, 
the following recommendations are made to improve DoD Financial 
Management Education and Training at all institutions in the DoD 
system. 

1. All DoD FM education and training institutions should employ 
student evaluation administered at the mid-point and at end of the 
course of study to provide feedback as to the quality and adequacy 
of instruction. Where such evaluational methods exist, 
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institutions should evaluate the instruments used to collect 
information from students to determine whether the necessary and 
correct information is being solicited from students. Few 
institutions presently administer mid-point evaluations presently. 
This deficiency should be corrected. 

2. Careful analysis of student course evaluations collected over 
the past three years, with emphasis on the most recent data, should 
be performed to determine changes needed in instruction. 

3. All DoD FM education and training institutions should develop 
and implement a system for tracking students that have received 
instruction. This system should continue to track students as 
their employment location changes. 

4. All DoD FM education and training institutions should develop 
and implement post-study student satisfaction surveys to obtain 
information from students regarding the usefulness of the education 
and training they received, as well as recommendations for change. 
Methods employed by selected institutions at present to collect 
these data should be assessed to determine the best approach to 
collection and analysis of student responses. Data should be 
collected and analyzed after the first year following the end of 
instruction and then subsequently in a three year cycle. 

5. All DoD FM education and training institutions should develop 
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and implement post-study employer satisfaction surveys to obtain 
information from the employers of graduated students regarding the 
usefulness to the employer of the education and training, as well 
as recommendations for change. Methods employed by selected 
institutions at present to collect these data should be assessed to 
determine the best approach to gather employer responses. Data 
should be collected and analyzed after the first year following the 
end of instruction and then subsequently in a three year cycle. 

6. As noted above, no single ,! best method" is feasible to assure 
quality in DoD financial management education and training. 
However, a systematic approach to quality assessment should be 
developed within DoD to assess quality on a regular basis to 
provide feedback to institutions and DoD executives on the quality 
and adequacy of instruction provided. Accordingly, a single FM 
education and training accreditation organization should be created 
to perform accreditation reviews of institutions providing FM 
instruction. Accreditation standards, guidelines and methods for 
evaluating compliance by institutions with these standards should 
be developed. A schedule for regular and periodic accreditation 
review including site visitation and participant observation should 
be established by this organization. 

7. Quality assurance accreditation reviews should be performed by 
the designated FM education and training accreditation organization 
assisted by subject matter specialists on a three year cycle to 
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assist institutions in maintaining currency in their curricula, 
instructional methods and technology. 

8. Accreditation reviews of institutions should be performed by 
teams consisting of one member of the accrediting agency and three 
subject matter specialists chosen from a pool of experts 
established for purposes of such review. Selection of subject 
matter experts should be based upon academic and professional 
experience and accomplishment in the field of instruction subject 
to review. The pool of experts should be drawn from within DoD and 
from institutions external to DoD. All accreditation teams should 
contain at least one expert from a non-DoD institution. To avoid 
duplicative review, where institutions already have received 
accreditation by recognized national or regional accreditation 
agencies, the DoD accreditation review agency may choose to accept 
such reviews in lieu of performing its own review. However, where 
this is the case, the results of the accreditation review performed 
by an external organization should be examined by the DoD 
accreditation agent to assure general conformance with DoD quality 
assurance standards and guidelines. 

9. The quality assurance accreditation compliance process should 
be designed to include the careful analysis of student course 
evaluations and information from the employers of graduated 
students regarding the usefulness to the employer of the education 
and training. 
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10. The quality assurance accreditation compliance process should 
be designed to assist institutions in meeting standard rather than 
punishing them for non-compliance. Resource augmentation to 
enhance library services, instructional technology and facilities 
should be considered where such aid appears to be needed. However, 
institutions that fail to implement and report on methods and 
actions to achieve compliance with DoD standards and guidelines 
within a reasonable period of time should be considered for 
budgetary sanction and possible termination. 

11. Based upon a review of the missions of DoD financial 
management education and training organizations, the quality 
assurance accreditation process should be developed and 
administered by the Defense Resource Management Institute advised 
by a committee that includes a single member representing each DoD 
FM education and training institution plus selected subject matter 
specialists in financial management education and training. This 
committee should meet semi-annually or on an "as needed" basis to 
provide guidance to the accreditation process. 

12. Based on survey results there is a need to establish or 
enhance financial management library resources at selected 
institutions. This need should be addressed as a part of the 
accreditation review process outlined above. 

13. Based on survey results there is a need to establish or 
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enhance computer and education technology resources at selected 
institutions. This need also should be addressed as a part of the 
accreditation review process outlined above. 

14. Based on survey results, there is a need to establish or 
enhance resources to support guest and visiting instructors and 
lecturers and speakers at selected institutions. This need should 
be addressed as a part of the accreditation review process outlined 
above . 

15. A pool of funds should be created and administered by the DoD 
accreditation agency to award annual grants on a competitive basis 
to institutions to establish or enhance library, computer, guest 
speaker/lecturer and other services to increase the quality of 
financial management education and training. Additionally, a 
competition should be established for an award for outstanding 
educational achievement by a DoD financial management education and 
training institution to recognize and reward quality performance. 
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APPENDIX A 



SURVEY METHODOLOGY 

The survey that was sent out to all providers of financial 
management education and training programs is included in Appendix 
B. The list of DoD agencies and departments responding to the 
survey is provided in Appendix C. 

Prior to developing and mailing the survey to all providers of 
financial management education and training, a comprehensive list 
of these providers needed to be verified. 

The principal sources for determining the agencies and departments 
to be surveyed were: 

(1) Formal Schools Directory. Third Edition [Ref. 8:] dated 
January, 1992. This directory was prepared by the DoD Training and 
Performance Data Center (TPDC) which is located in Orlando, 
Florida. The directory lists all schools and training locations 
operated by the Military Services and DoD agencies, including those 
identified as financial management. Five hundred eighty nine 
training organizations are identified in this directory. 

(2) Financial Management Data System [Ref. 9:] also 
developed by TPDC. The Financial Management Data System is an 
automated data collection system which is designed to facilitate 
the collection of descriptive and resource data for DoD financial 
management training and education courses. TPDC provided this 
researcher with the latest edition of the data base which was 
updated through January, 1992. 

The financial management data base was cross-referenced to the 
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Formal Schools Directory [Ref. 8:] to ensure that the latest 
available data on providers of financial management education and 
training was utilized. 

A final check was performed to ensure that all providers of 
financial management education and training were included. This 
included correspondence and communication with experts both at the 
Naval Postgraduate School and with field personnel in the various 
military departments. The final product or listing of providers 
was then prepared. 

DEVELOPMENT OF THE SURVEY INSTRUMENT 

The development of the survey used to gather data for this 
research began with an extensive review of the literature on survey 
design. [See references 10 through 21] Since the full spectrum of 
assessment techniques employed by DoD financial management 
education and training programs was not known, the basic 
methodological approach employed in the development of the 
questionnaire was the "open" response format. This format allows 
the recipient to reply to the questions in a manner best suited to 
the institutional setting of the respondent. [Ref. 11: p. 54] 
Additionally, filtering questions were developed to facilitate the 
response to questions that did not apply to certain recipients but 
did apply in general. Finally, general purpose or information 
questions were included in the question data base. 

A second methodological issue that was addressed was who 
should be contacted in the survey process. Mr. Peter Ewell, Senior 
Associate with the National Center for Higher Education Management 
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Systems (NCHEMS) noted that quality assessment of education could 
be conducted in two principal ways. These methods are: 

(1) Collection from the agencies and departments themselves 
providing an internal perspective, and 

(2) Collection of information from the clients served by the 
agencies and departments. 

The clients are either the students themselves or the 
sponsoring agencies which subsequently received the students. This 
method of data gathering is referred to as external. [Ref. 22] 
Due to the limited time frame and the prospect of uncertain 
response from the external sources, the internal method was 
selected. 

The third methodological issue addressed was the data 
collection method to be employed in the survey. Given the time 
constraints as well as the perceived length of the questionnaire on 
financial management quality assessment techniques, a combination 
of telephone and mail survey techniques were selected. In addition 
to the probability of gaining higher reliability of the responses, 
this method allowed the survey respondent to include additionally 
materials to the basic survey document. 

Initial Development of the Survey Questions 

Once the decision had been made by DoD Comptroller staff to 
perform a quality assessment of financial management education and 
training courses and programs, the initial development of the 
survey instrument was undertaken. In this phase of development, 
two steps were followed. The first step was to review research 
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literature on survey methodology. (Detailed in the previous 
section) . 

The second step was to contact professionals in the field of 
educational assessment to provide additional guidance on the survey 
design. 

One of the more significant contributions at this stage was 
provided by Peter T. Ewell, Senior Associate at the National Center 
for Higher Education Management Systems located in Boulder, 
Colorado. In his response to a request for assistance in 
developing a comprehensive survey, Mr. Ewell recommended the 
following procedures be assessed: 



• The nature of the program with respect to target audience 
(e.g., civilian/military, rank, geographical region) and its 
intensity/duration (e.g., one day, two-week on site, one year 
part-time, correspondence course) . 

• The learning objectives of the program, framed if possible in 
terms of "expected outcomes" of instruction i.e., particular 
elements of knowledge or skill that a student will exhibit at 
the programs conclusion. 

• The particular methods or modes of instruction used in 
delivering the program (e.g., hands-on training, simulation, 
classroom work, independent study, etc.), and in particular 
the ways in which students' competency is tested at its 
conclusion (e.g., paper-and-pencil exams, problems, rated 
demonstrations of hands-on performance, etc.). 

• How the effectiveness of the program in attaining its training 
objectives has been evaluated in the past; if possible, units 
should be requested to attach copies of any studies or data on 
effectiveness that they routinely collect to monitor and 
improve their own performance. [Ref. 22) 



Field Validation of the Survey Questions 

Once the initial set of survey questions had been compiled, 
the "validity" and "relevance" of these questions needed to be 
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tested. To accomplish this testing, selected agencies and 
departments offering financial management education and training 
programs were contacted through a telephone survey. The 
methodology employed in this telephone survey was ’’blind" response, 
i.e., the respondent was asked general questions on quality 
assessment and then asked what questions they would consider 
relevant in developing a survey instrument for their facility. Ten 
of the 31 institutions previously identified as providers of 
financial management education and training were contacted in this 
process. The providers were selected on the basis of size, service 
branch and referral from previously contacted providers. The 
various questions, suggestions and pertinent information collected 
during the telephone survey were added to the data base of 
questions prepared prior to the telephone survey. Appendix A lists 
those agencies and departments contacted during this phase of 
survey preparation. 

The Initial Survey Instrument 

Once the agencies and departments were selected for survey and 
the survey questions to be asked had been collected and developed, 
the next phase was to organize the survey questions into logical 
units. Specifically, the questions that had been obtained through 
the research noted previously had to be organized into a cohesive 
survey document. The general categories and principle questions or 
information requested is listed below: 

1 . General organizational and program information including : 
respondent, point of contact, target audience, number of 
financial management courses and programs offered, length 
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of the program, numbers of times offered, annual enrollment 
and average class size among others. 



2. Program requirements and development : How is the need 

determined for new courses? How is the need to update or 
modify courses determined? Is course development guided by 
higher authority? and Does the agency or department have 
staff dedicated to program development? 

3. Instruction : What policies are used to hire 

faculty/instructors? What methods are use to evaluate 
instruction and instructors? 

4 . Ongoing evaluation : What methods are used to determine the 

need for the current program? What methods are used to 
determine student competency? Are formal methods used for 
student course evaluation? and What percentage of students 
successfully complete the program? 

5. Program assessment : Is the performance of graduates 

tracked? What formal methods are employed to solicit 
feedback from former students and clients of those 
students? 



Once the questions were organized as noted above, the complete 
survey questionnaire was ready for initial pilot testing. The 
survey was given to CDR Glenn Eberling who taught the Navy 
Practical Controllership Course at the Naval Postgraduate School in 
Monterey, California and to Mr. Wade Cliendienst who was the Chief 
of Training and Professional Development for the US Army Audit 
Agency located in Alexandria, Virginia. 

Both recipients reviewed the survey and offered several 
additions to the proposed survey instrument. These additions 
focused on the methods of quality assessment and the organization 
of the survey. Both pilot recipients also suggested clarification 
in the wording of the questions and of the explanatory paragraphs 
included in the survey. 

A draft of the survey was reviewed by DoD Comptroller staff 
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and a cover letter from DoD Comptroller Sean O'Keefe was provided 
to introduce the survey. Sponsorship of the survey by the 
Comptroller was essential to ensure support from the agencies and 
departments surveyed. The cover letter from the DoD Comptroller is 
included at the end of the Appendix B. Of the 31 surveys sent to 
providers of financial management education and training, 21 were 
returned. The 21 responses were received prior to April 30,1992, 
the cut-off date for inclusion in the research project. This 
represents a response rate of €8 percent. Surveys returned by the 
Center for Army Leadership and the Army's Judge Advocate General's 
School indicated that neither agency offered financial management 
education and training. 

The response rate based on a revised total of 29 providers 
(the original thirty-one less the two mentioned above) with 19 
responses is 67 percent 1 . 

Another measurement of the response is to calculate the number 
of financial management programs managed by agencies and 
departments who responded to the survey. By this reckoning, the 



1 An effort was made to obtain a 100 percent response rate to 
the survey. Three non-responding agencies were specifically 
identified for intensive follow-up due to either their size or 
number of course offerings. They were the DLA Finance Training 
Section, the Defense Systems Management College and the Army Audit 
Agency. As a result of these efforts, surveys from the Defense 
Systems Management College and DLA Finance Training Section were 
received after the primary research on the thesis had been 
completed. The responses from these two agencies conformed to the 
results obtained from the nineteen survey respondents noted in this 
section. Also, Don Cress of the Army Audit Agency reported that he 
intended to complete and return the survey. A revised response 
rate based on 22 responses is 76 percent. The other non-responding 
agencies were contacted if the point of contact was known, but no 
other follow-up was conducted. 
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total number of financial management programs offered totals 181. 
Respondents to the survey manage 134 of the 181 programs. The 
response rate based on this measure is 79 percent 2 . 

Tables V(A) and V(B) detail the survey questions asked of 
respondents and the actual number of responses provided to each 
question. The variability in responses to the questions is based 
on (1) the non-applicability of certain question to the various 
respondents and (2) the failure of certain respondents to answer 
specific questions. The variability in response rate effects the 
discussion which follows. 



2 Including DLA, Defense Systems Management College and the 
Army Audit Agency, the response rate is 87 percent? 158 of 181 
programs . 
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TABLE V(A) : SURVEY QUESTIONS /NUMBER OF RESPONSES 



SURVEY QUESTION 


NO. OF RESPONSES 


1. 


TARGET AUDIENCE 


19 


2 v . 


PERCENTAGE OF CLASS ATTENDEES 


19 


3. 


GRADES/RANKS OF ATTENDEES 


19 


4. 


NUMBER OF COURSES 


19 


5. 


LENGTH 


17 


6. 


TIMES OFFERED 


15 


7. 


ANNUAL ENROLLMENT 


17 


8 . 


AVERAGE CLASS SIZE 


19 


9. 


COLLEGE CREDITS 


8 


10. 


DEGREE OR CERTIFICATE OFFERED 


19 


11. 


HOW DETERMINE NEED FOR NEW COURSES 


18 


12. 


HOW DETERMINE NEED TO UPDATE COURSES 


17 


13. 


SPONSORS 


19 


14. 


COURSE DEVELOPMENT GUIDED 


19 


15. 


TYPES OF GUIDANCE RECEIVED 


14 


16. 


HOW OFTEN IS GUIDANCE RECEIVED 


17 


17. 


STAFF DEDICATED TO PROGRAM DEVELOPMENT 


18 


18. 


PERCENTAGE OF MIL. /CIV. INSTRUCTORS 


16 


19. 


POLICIES/CRITERIA TO HIRE MILITARY 


11 


20. 


POLICIES/CRITERIA TO HIRE CIVILIANS 


10 


21. 


PROFESSIONAL QUALIFICATION STANDARDS 


18 


22. 


CLASSROOM OBSERVATION 


19 


23. 


OTHER METHODS TO EVALUATE INSTRUCTION 


18 


24. 


OTHER METHODS TO EVALUATE INSTRUCTORS 


18 
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TABLE VfB) : BURVEY QUESTIONS /NUMBER OF RESPONSES 



SURVEY QUESTIONS 


NO. OF RESPONSES 


25. 


PRODUCTIVITY MEASURES 


19 


26. 


INSTITUTION ACCREDITED 


19 


27. 


INSTITUTION REVIEWED 


19 


28. 


NATURE OF REVIEW 


9 


29. 


METHODS OF INSTRUCTION 


19 


30. 


PERCENTAGE OF INSTRUCTION BY METHOD 


11 


31. 


TYPES OF COURSE READING MATERIAL 


16 


32 . 


LIBRARY SUPPORT 


18 


33. 


IMPROVE LIBRARY SUPPORT 


6 


34. 


IMPROVEMENTS PLANNED IN LIBRARY 


17 


35. 


COMPUTER SUPPORT 


19 


36. 


COMPUTER EXERCISES 


18 


37. 


PROGRAMMATIC SUPPORT 


19 


38. 


METHODS TO DETERMINE NEED FOR PROGRAM 


18 


39. 


METHODS TO ASSESS VALIDITY, RELEVANCE 


19 


40. 


METHODS TO DETERMINE COMPETENCE 


16 


41. 


FORMAL METHOD OF STUDENT EVALUATION 


19 


42. 


OTHER METHODS FOR STUDENT EVALUATION 


19 


43. 


PASS STANDARDIZED EXAMS OR TESTS 


19 


44 . 


PERCENTAGE OF STUDENTS COMPLETING 


11 


45. 


GRADUATE PERFORMANCED TRACKED 


19 


46. 


FEEDBACK FROM FORMER STUDENTS 


19 


47. 


MOST IMPORTANT METHODS OF QUALITY 


19 
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APPENDIX B: 



DEPARTMENT OF DEFENSE FINANCIAL MANAGEMENT EDUCATION 

AND TRAINING QUALITY ASSESSMENT SURVEY 

PURPOSE: This survey questionnaire is designed to determine how 
DoD financial management education and training programs assess the 
quality of their programs. Quality in the context of this survey 
means providing accurate, valid, comprehensive and up-to-date 
information to meet the needs of your customers or clients. 
Quality begins with the determination of the need for the program 
and continues through the monitoring of the performance of 
graduates. Quality consists of providing the correct instruction 
for the appropriate target population. 

The following series of questions is designed to allow you to 
provide information on quality assessment techniques employed by 
your organization. We encourage you to answer the questions 
completely and add additional information wherever appropriate. 
Please attach additional pages and materials as necessary to 
provide complete answers. 



ORGANIZATIONAL INFORMATION: 
Respondent's Name: 



Position: 

Agency/Component Name: 
Address: 



Phone Number: 
(DSN/Autovon) 



Please provide the title or name of the instructional program 
offered at your institution: 



GENERAL PROGRAM INFORMATION : 

1. What is the target audience for your program? 

(CIVILIAN) (MILITARY) (BOTH) 

2. What are the approximate percentages of class attendees? 

CIVILIAN: % 

MILITARY: % 

3. What are the ranges of grades or ranks of your attendees? 

CIVILIAN: 

(GS or GM - to GS or GM - ; SES Level) 

MILITARY: 

4. How many separate courses in financial management are 
offered within your program? 



5. What is the length of your program in days, weeks or 

months? 

6. How many times per year is it offered? 

7. What is the annual enrollment in your program? 



8. What is the average (mean) class size? 



9 . 



If your program provides course credit for college, how many 
credits are offered in the total program? 



10. Do students completing the program receive a degree or 
certificate? 

(YES) (NO) 

If yes, what is the title of this award? 



The rest of the questionnaire is organized into the following 
parts: (1) Program requirements and development (2) Instruction (3) 
Ongoing evaluation and (4) Program assessment. Some of the 
questions require a simple YES/NO response, other ask for short 
responses, helpful comments and recommendations. We ask you to 
carefully complete this survey so that we can develop a 
comprehensive data base of quality assessment measures employed in 
financial management education and training. 



PROGRAM REQUIREMENTS AND DEVELOPMENT: 

11. How do you determine need for new courses or instruction? 



12. How do you determine the need to update or modify the 
instruction you currently offer? 



13. Do you have one or more sponsors or clients that directly 
influence your program of instruction? 



(YES) 



(NO) 



If yes, who is/are the sponsor (s)? 



14. Is course development guided or directed by higher 
authority? 

(YES) (NO) 

If so, 

who? 

15. What types of guidance do you receive? 



16. How often do you receive such guidance? 



17. Do you have staff dedicated to program development? 
(YES) (NO) 

If yes, how many? 



INSTRUCTION: 

18. What are the number of and percentages of civilian and/or 
military instructors in your program? 

CIVILIAN: Number Percentage 

MILITARY: Number Percentage 



19. What policies and criteria are used to assign or hire 

military faculty/instructors for your program? Please attach 
any material or guidance that describes these policies or 
criteria to your survey response. 



20. What policies and criteria are used to appoint or hire 

civilian faculty/ instructors for your program? Please attach 
any material or guidance that describes these policies and 
criteria to your survey response. 



21. Are professional qualifications standards required for 
faculty/ instructors (e.g., degrees, prior experience, 
completion of an instructor training program) ? 

(YES) (NO) 

If yes, please describe. Attach written guidance or 
material that describes such qualifications to your survey 
response. 



22. Do you use classroom observation to evaluate instruction? 
(YES) (NO) 

If yes, who performs this observation? 



How often is it performed? 



23. Are other methods used to evaluate classroom instruction? 
(YES) (NO) 

If yes, please describe: 



24. Are other methods used to evaluate instructors? 



(YES) (NO) 

If yes, please describe: 



25. Do you employ productivity measures to evaluate instructors? 
(YES) (NO) 

If so, please describe. Attach any materials that 
describe this process to your survey response. 



26. Is your institution accredited? 

(YES) (NO) 

If yes, by whom? 

How often is the accreditation review performed? 



27. Is your institution reviewed by other agencies? 
(YES) (NO) 

If yes, by whom and how often? 



28. What is the nature and extent of this review? Please attach 
any materials that describes this process to your survey 
response. 



29. What methods or modes of instruction are utilized in your 

program (e.g., lecture, discussion, simulation, independent 
study) ? 



30. What percentage of instruction is delivered in each mode you 
identified? 



31. What types of course reading materials are used in your 

program (e,g., textbooks, published articles, DoD official 
documents, self-generated materials)? 



32. Is your program supported by a library containing extensive 
DoD financial management related materials? 

(YES) (NO) 

33. How would you like to improve library support for your 
program? 



34. Are there improvements in your library services planned? 
(YES) (NO) 

If yes, please explain these improvements and indicate 
when they will occur. 



35. Is your program supported by a computer center or 
laboratory? 

(YES) (NO) 

If no, what computer resources are employed at your 
institution? 



36. Does your instructional program include computer exercises 
or assignments? 

(YES) (NO) 

If yes, please describe how computers are utilized. 



37. Is your program well supported with respect to 

administrative staff, funding for guest speakers, etc.? 

(YES) (NO) 

What improvements in support would be desirable? 



ONGOING EVALUATION: 

38. What methods are used to determine the need for your current 
program? 



39. What methods are used to determine the validity, accuracy, 
relevance and timeliness of your current program? 



40. What methods are employed to determine student competence 
during and upon program completion (e.g., passing 
standardized tests, written course work, observation of 
performance on the job)? 



41. Do you have a formal method of student course evaluation? 
(YES) (NO) 

If yes, describe this process and provide the form used 
to gather student evaluations with your response. 



42. Are other methods used to permit students to evaluate your 
instructional program? 

(YES) (NO) 

If yes, describe these methods: 



43. Are your students required to pass any standardized DoD or 
national examinations (e.g., CPA, CMA) ? 



(YES) 



(NO) 



If yes, please 

identify: 

44. What percentage of students entering your program 

successfully complete it? % 

PROGRAM ASSESSMENT: 

45. Do you track the performance of your graduates? 

(YES) (NO) 

If yes, how is this done? 



46. Are formal methods employed to solicit feedback from former 
students and clients about the utility of your program? 

(YES) (NO) 

If yes, please explain how this is done and provide 
examples of such feedback. 



47. In your opinion, what are the most important methods you 
employ to ensure the quality of your program? 



48. Is there any other information that you would offer on 
quality assessment at your institution that we did not 
request? Please provide comments as appropriate. 



APPENDIX C 



DoD FINANCIAL MANAGEMENT AGENCIES AND DEPARTMENTS 
RESPONDING TO THE SURVEY 

ARMY FINANCIAL MANAGEMENT SCHOOLS : 



US Army Finance School 

Fort Benjamin Harrison, IN 46216-5640 

US Army Management Engineering College 
Rock Island Arsenal, IL 61229-7040 

Syracuse University 
Army Programs Office 
310 School of Management 
Syracuse, NY 13244-2130 

US Army Engineer Division 
Directorate of CE Training Management 
Huntsville, AL 35807-4301 

US Army Logistics Management College 
Fort Lee, VA 23801-6056 

* The Judge Advocate General's School 
Contract Law Division 
Charlottesville, VA 22903-1781 

* Center for Army Leadership 
Fort Leavenworth, KS 66027 



AIR FORCE FINANCIAL MANAGEMENT SCHOOLS : 



3750TH Technical Training Group 
Sheppard Air Force Base, TX 76311-5434 

Air Force Institute of Technology 
Wright-Patterson Air Force Base, OH 45433-6583 

Air University 

Professional Military Comptroller School 



Maxwell Air Force Base, AL 36112-5712 



NAVY FINANCIAL MANAGEMENT SCHOOLS : 



Naval Postgraduate School 
Department of Administrative Sciences 
Monterey, CA. 93943-5022 

American University 
Washington, DC 20016-8070 

Troy State University 

P. O. Box 1032 

Fort Meyer, VA 22211 

Navy Comptroller 

Program Management Office 

Naval Air Station Pensacola 

Building 625D 

Pensacola, FL 32508-5175 

Fleet Training Center 
Norfolk, VA 23511 

Civil Engineer Corps Officers School 
Port Hueneme, CA 93043-5002 

Naval School of Health Sciences 
Bethesda, MD 20814-5033 

Fleet Aviation Specialized Operational Training Group 
Norfolk, VA 23511 

Navy School of Manpower Management 
Norfolk, VA 23511 

Naval Military Personnel Command 
Navy MWR Recreation Training Unit 
Patuxent River, MD 20670 



MARINE CORPS FINANCIAL MANAGEMENT SCHOOLS : 

Financial Management School 
Marine Corps Service Support Schools 
Camp Lejeune, NC 28542-5050 



MOTE: An asterisks preceding the name indicates that a 
response was received. However, the response was that the 
agency or department did not perform financial management 
education and training. 
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MEMORANDUM FOR ASSISTANT SECRETARY OF THE ARMY (FINANCIAL 

MANAGEMENT) 

ASSISTANT SECRETARY OF THE NAVY (FINANCIAL 
MANAGEMENT) 

ASSISTANT SECRETARY OF THE AIR FORCE (FINANCIAL 
MANAGEMENT AND COMPTROLLER) 

DIRECTOR OF THE JOINT STAFF 

SUBJECT: Financial Management Review 



Within the next couple of months you may be contacted by 
Professor Larry R. Jones from the Naval Postgraduate School 
requesting interviews with you and your staff concerning the 
relationship between the Defense Management Report (DMR) and DOD 
financial management. 

Among the benefits that I expect from Professor Jones' 
efforts are: 

• an academic review of the DMR process on financial 
management within DOD, 

• an evaluation of financial management changes 
undertaken within the DMR process, 

• some insights into the role financial management 
education resources should play into the DMR process, 
and 

• some insights into the future course of financial 
management with respect to the Defense Management 
Report. 

Your cooperation with Professor Jones will be beneficial to 
us all and greatly appreciated. 



Sean O'Kee 
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